A Survey of Bitmap Index-Compression Algorithms for Big Data

With the growing popularity of Internet applications and the widespread use of mobile Internet, Internet traffic has maintained rapid growth over the past two decades. Internet Traffic Archival Systems (ITAS) for packets or flow records have become more and more widely used in network monitoring, network troubleshooting, and user behavior and experience analysis. In this paper, we survey bitmap-index compression algorithms for traffic archival systems. The current state-of-the-art bitmap-index encoding schemes include: BBC, WAH, PLWAH, EWAH, PWAH, CONCISE, and COMPAX. Based on differences in segmentation, chunking, merge compress, and Near Identical (NI) features, we provide a thorough categorization of the state-of-the-art bitmap compression algorithms. We also propose some new bitmap encoding algorithms-SECOMPAX, ICX, MASC, PLWAH+-and show the state diagrams for their encoding algorithms. We then evaluate their CPU and GPU implementations with a real Internet trace from CAIDA. Finally, we summarize and discuss the future direction of bitmap-index compression algorithms.